Media Content Identification on Mobile Devices

ABSTRACT

A mobile device responds in real time to media content presented on a media device, such as a television. The mobile device captures temporal fragments of audio-video content on its microphone, camera, or both and generates corresponding audio-video query fingerprints. The query fingerprints are transmitted to a search server located remotely or used with a search function on the mobile device for content search and identification. Audio features are extracted and audio signal global onset detection is used for input audio frame alignment. Additional audio feature signatures are generated from local audio frame onsets, audio frame frequency domain entropy, and maximum change in the spectral coefficients. Video frames are analyzed to find a television screen in the frames, and a detected active television quadrilateral is used to generate video fingerprints to be combined with audio fingerprints for more reliable content identification.

The present application is a continuation of U.S. patent applicationSer. No. 15/053,064, filed Feb. 25, 2016, which is a divisional of U.S.patent application Ser. No. 13/590,701, filed Aug. 21, 2012 which claimsthe benefit of U.S. Provisional Patent Application No. 61/601,234entitled “Methods and Apparatus for Synchronous Television/Media ContentIdentification on Mobile/Media Devices”, filed on Feb. 21, 2012 whichare hereby incorporated by reference in their entirety.

CROSS REFERENCE TO RELATED APPLICATION

U.S. Pat. No. 8,229,227 filed on Jun. 18, 2008 entitled “Methods andApparatus for Providing a Scalable Identification of Digital VideoSequences”, U.S. Pat. No. 8,171,030 filed on Jun. 18, 2008 entitled“Method and Apparatus for Multi-Dimensional Content Search and VideoIdentification”, U.S. Pat. No. 8,189,945 filed on Nov. 5, 2009 entitled“Digital Video Content Fingerprinting Based on Scale Invariant InterestRegion Detection with an Array of Anisotropic Filters”, U.S. Pat. No.8,195,689 filed on May 3, 2010 entitled “Media Fingerprinting andIdentification System”, U.S. patent application Ser. No. 13/094,158filed on Apr. 26, 2011 entitled “Actionable Event Detection for EnhancedTelevision Delivery and Ad Monitoring Based on Video/Audio ContentFingerprinting”, and U.S. Provisional Patent Application Ser. No.61/610,672 filed on Mar. 14, 2012 entitled “A Method for Efficient DataBase Formation and Search on Portable Media Devices Acting Synchronouslywith Television Programming”, have the same assignee as the presentapplication, are related applications and are hereby incorporated byreference in their entirety.

FIELD OF THE INVENTION

The present invention relates to identifying television or other mediaprogramming at a receiving station by using a mobile device. Moreparticularly, the present invention addresses design of an efficienttelevision/media identification system based on fingerprinting ofcaptured audio and video signals in the presence of ambient noise,including speech and music interference from multiple external sources,as well as various optical and geometry distortions of the video signal.

BACKGROUND OF THE INVENTION

Recent development of audio and video content fingerprintingtechnologies and capable mobile devices, such as smart phones andtablets, have opened up a plethora of new possibilities in developingsophisticated real time applications associated with detected televisionprogramming events. With the ubiquity of mobile devices, especiallysmart mobile phones, a large proportion of the population oftensimultaneously watch programming content on their television while usingtheir portable mobile device for text messaging or other Internetrelated activities. Due to the presence of varying levels of ambientnoise and image distortions, reliably identifying content that is beingplayed on a television set is considered a difficult capability toprovide on a portable mobile device. Such capability has to be robust topotential audio and video degradation in order to accurately process andidentify audio and video content.

SUMMARY OF THE INVENTION

In one or more of its several embodiments, the invention addressesproblems such as those outlined in the previous section. One embodimentof the invention addresses a method for a mobile device to respond inreal time to content identified on a television program display device.Audio content generated on a television (TV) display device is capturedas a waveform from a microphone on the mobile device. Video contentdisplayed on the TV display device is captured as a set of video framesfrom an optical image capture device on the mobile device.

Contours of a TV display screen on the TV display device are detected inone or more video frames on the mobile device, wherein the detectedcontours of the TV display screen are overlaid on images of the capturedvideo content displayed on the mobile device.

Another embodiment addresses a method for audio fingerprinting by usingcontent based audio feature extraction. Input audio samples, dividedinto overlapping frames, are analyzed to produce windowed audio framesamples for each overlapped frame. A fast Fourier transform (FFT) forthe windowed audio frame samples is computed which FFT results arefiltered by a filter bank on the spectral components to produce a set offilter bank output coefficients. A log function and a square rootfunction of each set of filter bank output coefficients are computed toproduce log and square root output coefficients. A discrete cosinetransform (DCT) is applied separately to the log and square root outputcoefficients to produce two sets of DCT spectral components. A temporalmulti-tap finite impulse response (FIR) smoothing derivative filter isapplied to the two sets of DCT spectral components to produce twoseparate primary descriptors, wherein values in the two separate primarydescriptors are sampled to produce two primary signatures.

Another embodiment addresses a method for audio content featureextraction. An onset in a time domain is detected for each audio frameof a plurality of audio frames. A frequency domain entropy is calculatedfor each audio frame of the plurality of audio frames to produce anentropy difference between consecutive frames. A maximum difference inthe spectral output coefficients is calculated for each audio frame ofthe plurality of audio frames.

Another embodiment of the invention addresses a method for audio signalonset detection and audio frame time positions for alignment based ondetected audio signal onsets. A multi-channel audio signal is down mixedto a mono signal and resampled to a desired sampling rate. An energywaveform of the audio signal is computed by squaring the audio waveform.A low-pass filter is applied to the energy signal and resampled to aminimum sampling period. A filtered derivative of the resultingresampled signal is computed for different filter widths. Maximumindices of the filtered derivative signal are computed for differentmaximum filter widths to produce time positions of maximum indices thatexceed a threshold, wherein the time positions represent onset events.

Another embodiment addresses a method to enable mobile device softwareapplications to provide a real time response to an identified segment ofbroadcast television media content. Audio content and video content arecaptured on a mobile device. On the mobile device, multi-dimensionalaudio and video query signatures and multi-dimensional featuresignatures are generated for audio and video features identified in atemporal segment of audio and video data received on the mobile device.On the mobile device, cluster query signatures are generated based on acombination of the multi-dimensional audio and video query signaturesand the multi-dimensional feature signatures. A reference multimediaclip database is searched, as initiated by the mobile device, using themulti-dimensional cluster query signature for fast reference data basetraversal to find a set of signatures that are within a specifiedsignature distance to the multi-dimensional query signature, wherein themobile device is provided access to data related to multimedia contentassociated with a likely matching signature selected from the set ofsignatures. Based on the search results, a software application istriggered within the mobile device, which performs at least one actionthat is synchronized to the identified captured audio and video content.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a system for providing mobile device applications inaccordance with the present invention based on analysis of audio, video,or audio and video signals acquired from a media presentation device;

FIG. 2 illustrates a media acquisition subsystem configured to acquireplayed television media content from a television device on a mobiledevice;

FIG. 3 illustrates a process for identification of audio or videocontent captured by a mobile device from a nearby TV set and triggeringa mobile application action;

FIG. 4A illustrates a process for processing a video fragment acquiredby a mobile camcorder operating on the smart phone;

FIG. 4B illustrates a process for multiple primary audio signaturegeneration;

FIG. 5 illustrates a process for cluster signature generation;

FIG. 6 illustrates a process for audio signal onset detection in thetime domain;

FIG. 7 illustrates a process for utilization of audio signal onsetinformation;

FIG. 8 illustrates a process for onset detection in the time domainbased on audio frame envelope tracking and continuous moving averagetracking as an audio frame feature;

FIG. 9 illustrates a process for audio frame frequency domain entropycomputation as an audio frame feature;

FIG. 10 illustrates a process to compute position of the maximum changein the audio descriptor coefficients between two consecutive audioframes, which is used as an audio frame feature;

FIG. 11A illustrates an exemplary signature data structure used in asignature database;

FIG. 11B illustrates a signature selection and database formationprocess;

FIG. 12 illustrates a process for fingerprinting and searching withmultiple signatures and features;

FIG. 13 illustrates a process for combining audio and video data basesearch results to refine the search process and improve accuracy infinding a matching audio/video sequence;

FIG. 14 illustrates a system for actionable event detection inaccordance with an embodiment of the present invention;

FIG. 15 illustrates a system view of a reference point with a multiplesystem operators device, satellite device, and Internet protocol (IP)equipped TV set, and a remote monitoring point in accordance with anembodiment of the present invention;

FIG. 16 illustrates a process for audio and video fingerprinting atclient devices such as an STB, and an IP TV set, and referencetelevision content database formation in accordance with an embodimentof the present invention;

FIG. 17 illustrates a process for video snippet fingerprinting,preloaded reference advertisement database search, and actionable eventgeneration in accordance with an embodiment of the present invention;and

FIG. 18 illustrates a process for remote querying of the reference TVdatabase, and actionable event generation based on recently playedtelevision content identification in accordance with an embodiment ofthe present invention.

DETAILED DESCRIPTION

The present invention will now be described more fully with reference tothe accompanying drawings, in which several embodiments of the inventionare shown. This invention may, however, be embodied in various forms andshould not be construed as being limited to the embodiments set forthherein. Rather, these embodiments are provided so that this disclosurewill be thorough and complete, and will fully convey the scope of theinvention to those skilled in the art.

It will be appreciated that the present disclosure may be embodied asmethods, systems, or computer program products. Accordingly, the presentinventive concepts disclosed herein may take the form of a hardwareembodiment, a software embodiment or an embodiment combining softwareand hardware aspects. Furthermore, the present inventive conceptsdisclosed herein may take the form of a computer program product on acomputer-readable non-transitory storage medium having computer-usableprogram code embodied in the storage medium. Any suitable computerreadable non-transitory medium may be utilized including hard disks,CD-ROMs, optical storage devices, flash memories, or magnetic storagedevices.

Computer program code or software programs that are operated upon or forcarrying out operations according to the teachings of the invention maybe written in a high level programming language such as C, C++, JAVA®,Smalltalk, JavaScript , Visual Basic®, TSQL, Perl, use of .NET™Framework, Visual Studio® or in various other programming languages.Software programs may also be written directly in a native assemblerlanguage for a target processor. A native assembler program usesinstruction mnemonic representations of machine level binaryinstructions. Program code or computer readable medium as used hereinrefers to code whose format is understandable by a processor. Softwareembodiments of the disclosure do not depend upon their implementationwith a particular programming language.

The methods described in connection with the embodiments disclosedherein may be embodied directly in hardware, in a software module thatstores non-transitory signals executed by a processor, or in acombination of the two. A software module may reside in RAM memory,flash memory, ROM memory, EPROM memory, EEPROM memory, registers, harddisk, a removable disk, a CD-ROM, or any other form of non-transitorystorage medium known in the art. A computer-readable non-transitorystorage medium may be coupled to the processor through local connectionssuch that the processor can read information from, and write informationto, the storage medium or through network connections such that theprocessor can download information from or upload information to thestorage medium. In the alternative, the storage medium may be integralto the processor.

FIG. 1 illustrates a system 100 for providing mobile device applicationsin accordance with the present invention based on analysis of audio,video, or audio and video signals acquired from a media presentationdevice. The input of audio, video, or audio and video signalshereinafter is referred to as media content. One embodiment addresses amethod to identify media content, by acquiring media content on a mobiledevice from a media presentation device, appropriately processing themedia content, fingerprinting the media content, and searchingfingerprint databases to identify the media content. The system 100includes a media presentation device 104, such as a television set, adesktop, laptop, or tablet computer, a digital video disc (DVD) player,or a smartphone device configured to display television programming orplay audio media such as wireless received radio signals, compact discs(CDs), or the like. The media presentation device, such as a televisionset, may be connected to a remote media broadcast system 105 forreceiving television programming content. Also, the media presentationdevice 104 may or may not be connected to the Internet 106. The system100 also includes a remote content identification system 108 and amobile device 110 connected over the Internet 106 to the remote contentidentification system 108. Connections 112, 114, and 116 may be wired orcabled connections, wireless connections, or a combination of wire cableand wireless connections. For example, the mobile device 110 may connectwirelessly to the Internet 106 through a wireless router or over 3G or4G networks while the remote content identification system 108 may bewire or cable attached through a modem to the Internet 106. The mobiledevice 110 may be configured to acquire audio signals from the mediapresentation device 104 by use of a microphone. The mobile device 110may also be configured to acquire images or a video from the mediapresentation device 104 by use of a charge coupled device (CCD) imagesensor and a camera lens assembly included in the mobile device 110.

Illustrated system 100 supports applications on the mobile device 110that operate in real time and in accordance with television or othermedia programming content that is being presented on a mediapresentation device 104 and received by the mobile device 110.

The mobile device 110 is configured to acquire a temporal fragment ofmedia content, including audio content, video content, or both, that areplaying on the media presentation device 104, using the mobile device'smicrophone, camera, or both, and generates query fingerprints of thetemporal fragment of the acquired media content. A chunk of the queryfingerprints, which is a set of the query fingerprints corresponding toa time segment of the query audio signal, or a digest of the chunk ofthe query fingerprints are transmitted as a search query to the remotecontent identification system 108, also referred to as a remote searchserver 108, for content identification. A digest of the queryfingerprints is a summarization of the fingerprints generated for theacquired media content. If the search query is found in a referencedatabase of the search server 108, the search server 108 responds with atitle and timing information of the identified media content, along withrelated metadata, and sends the title, the timing information, and therelated metadata to the mobile device 110. The original chunk of queryreference fingerprints or the digest of the query fingerprints is storedon the mobile device 110 for further use in querying a mobile devicedatabase located on the mobile device 110 and tracking of media content.The mobile device 110 may be configured to continuously listen, observe,or listen and observe the media programming content. If a change in themedia programming content is detected, the mobile device 110 generatesone or more new queries that are sent to the remote search server 108for content identification. If the new query is found in the referencedatabase of the remote search server 108, the search server 108 respondswith a title and timing of the media content associated with the newquery, along with related metadata, and sends the identified informationto the mobile device 110. The original new chunk of referencefingerprints are stored on the mobile device 110 for further use inquerying and tracking operations locally on the mobile device 110. Thisprocess continues as long as the mobile device 110 is listening, orobserving, or both to the media programming content. The mobile device110 may be equipped with an actionable program event detection system,which generates an action signal upon detection of a particular audio,or video, or audio and video fragment stored in the referencefingerprint database. A software application running on the mobiledevice 110 can then perform actions based on local search results,presenting to the user a variety of additional information on the samemobile device 110 in real time while the remote media programming isstill playing the associated media content.

For example, a movie that started at 9 PM is being watched on atelevision set 104. A user enables an application on a mobile device110, such as a smartphone, that configures the smartphone 110 to acquirea fragment of media content, which may include a fragment of audiocontent, a fragment of video content, or fragments of both audio andvideo content. For example, a fragment may be five seconds of backgroundmusic from a scene in the movie. A fragment may also be a snapshot of acharacter in the movie or may be a short narrative given by a leadingcharacter in the movie. If a video fragment is acquired by a mobilecamcorder or camera operating on the smartphone 110, video frames of thefragment are initially analyzed to find the TV screen in the frames. Inan exemplary case, the screen location step may be done by running edgedetection on selected frames, which may also include running contourdetection on the selected frames, combined with contour thresholding andselection, and searching for an initial quadrilateral of appropriatedimensions. A detected quadrilateral is further refined by trackingmotion from frame to frame of pixel formations inside and in theimmediate neighborhood of the quadrilateral. Also, brightness and colorof the detected quadrilateral can be checked against the rest of aframe's content to further increase confidence that the TV screen areais correctly delineated. The user may be informed that a TV screen isdetected by displaying an outline of a TV screen quadrilateral on thesmart phone display. If no TV screen is found, then the lack ofacquiring a TV screen may be communicated to the user by appropriatelychanging the appearance of virtual guidelines on the smartphone display,by making them flash, changing their color, and the like, for example.In the case of a detected TV screen, the frame area corresponding to thedetected quadrilateral is cropped and warped to an upright rectangle andused for video fingerprint generation of the TV programming contentcaptured from the TV set. Also, if the smart phone is held close to theTV, the video captured on the smartphone may be filled with content fromthe TV screen and a TV screen detection process would not be used. Insuch a case, the original captured frames are processed as holding TVprogramming content.

The application generates query fingerprints for the acquired fragmentof media content as described in U.S. Pat. Nos. 8,229,227, 8,171,030,8,189,945, and 8,195,689, and U.S. patent application Ser. No.13/094,158 which are incorporated by reference in their entirety. Theapplication transmits the query fingerprints as a search query to aremote search server 108 which searches for the transmitted content in areference database of the remote search server 108. If media contentassociated with the query fingerprints is discovered, the remote searchserver 108 retrieves related content, such as a title of the mediacontent, timing information and identifies other related informationwhich the user may have previously requested and forwards the relatedcontent and associated information to the user's smartphone 110 forpresentation to the user. At this point, the television programming istracked in real time and preprogrammed events are identified, such as achange in television program, or onset of a selected type of commercial,or entrance of a particular character, with sub-second resolution and totrigger a notification action to alert the user.

By using such a content identification system, it is possible toconfigure a real-time media content analysis software application, torun on the mobile device itself. FIG. 2 illustrates a media acquisitionsubsystem 200 configured to acquire played television media content froma television device 204 on a mobile device 206. The television device204 is configured to play a television program that generates sounds andspeech from speakers in synchronism with images and video from a displaydevice, such as a liquid crystal display (LCD) or light emitting diode(LED) display. The mobile device 206 is configured with a microphone 208to acquire the sounds and speech and transform the sounds and speech toaudio signals. The mobile device 206 is also configured with a cameradevice 210 that acquires the images and video, and transforms the imagesand video to a sequence of pixels or frames of pixels. The mobile device206 is configured with a central processing unit (CPU) system 212 thatacquires the audio and pixel data for analysis.

A technical concern in enabling mobile applications to operate in realtime with, for example, television content played on a nearby televisiondevice is to be able to accurately identify the media content acquireddirectly from the TV set by the mobile device's microphone, camera, orboth. Such acquisition operates in a dynamic environment of the mobiledevices which tends to degrade the quality of the content beingacquired. For example, the quality of an audio signal may be degraded bysources including lossy encoding of the source audio, fidelitylimitations of the speaker system, equalization, multi-path interferenceusing a multi-speaker system, fidelity limitations of the microphone onthe mobile device, automatic gain adjustments or equalization on thespeaker and/or microphone, and the encoding of the audio on the mobiledevice. With such degradations in the audio content, contentidentification based on the audio signal captured from a nearby TV setis a challenging problem. Even more severe signal degradation situationsmay arise with respect to the image and video pixel data acquired from anearby TV set. The sources of degradation are numerous, including theencoding of the source video, fidelity limitations of a display device,such as a the television screen, automatic brightness and contrastadjustments on the display device, the fidelity limitations of the videocamera on the mobile device, automatic brightness and contrastadjustments of the video camera on the mobile device, environmentallighting conditions, the viewing angle of the camera and any perspectivedistortion ensuing, and the encoding of the video on the mobile device.

FIG. 3 illustrates a process 300 for identification of audio or videocontent captured by a mobile device from a nearby TV set and triggeringa mobile application action. At step 304, a television program isselected to be played on a TV set. At step 306, a fragment of audio andvideo signals are acquired on the mobile device. The audio and videosignals are passed to the associated step in the mobile application forprocessing the audio and video signals as an audio waveform and videoframes, respectively. At step 308, the audio waveform is framed. At step310, the framed audio waveform is fingerprinted with an audiofingerprinting system. At step 312, one or more active regions aredetermined in the video frames and a set of frames are selected forfurther processing. At step 314, active TV regions in the selected videoframes are fingerprinted with a video fingerprinting system. Thisfingerprinting may occur entirely locally on the mobile device, entirelyremotely on a remote server, or partially locally and remotely. Ifperformed entirely remotely, the audio waveform and the video frames aretransmitted to the remote server. Alternatively, some partialfingerprint processing may be done locally and then the remainder of thefingerprint processing may be done remotely. In this alternativeembodiment, video frame selection on the video frames is performedlocally on the mobile device and then only the selected video frames aretransmitted to the remote server for fingerprint processing. For audio,the audio waveform may be re-sampled to a lower sampling rate and downmixed to one channel, before generating the fingerprints. Compression ofthe video frame and the audio samples may be used to reduce transmissioncosts.

At step 320, the audio fingerprints and video fingerprints are combinedand a selected set of fingerprints are used as query fingerprints.Having both audio fingerprints and video fingerprints representing theTV programming content increases the reliability of TV contentidentification under severe audio and video signal degradations due tothe surrounding ambient conditions. The resulting audio and video queryfingerprints are transmitted to a search server. A search function maybe either local, residing on the mobile device or remote, accessed forexample through the Internet cloud. At step 322, the search serverresponds with a message that details where the audio and video contentwere found in the search database, and if found, the title of thecontent, the matching times, and related metadata, like an imagerepresenting the program, details about actors, or the like. If a matchis not found at step 322, the process 300 returns to step 306 to selectanother fragment of media content for processing.

At step 324, the mobile application receives the match data and may beconfigured to trigger actions that are based on this data. Such actionsmay include displaying the identity of the content to the user,retrieving related information based on the identity of the content,allowing the user to register that piece of content with a registrationserver online, display an interactive ad based on the content andperhaps knowledge about the user, or may enable a variety of other realtime applications based on detected audio and video content.

One embodiment of the invention addresses a method for improving theaccuracy and speed of audio fingerprinting by using content based audiofeature extraction and signature generation. Audio features,representing the audio content, are extracted by using a mel-frequencycepstrum coefficients (MFCC) algorithmic structure with an additionaltemporal multi-tap filtering of the output coefficients, and subsequentgeneration of compact, bit-wise representation of these features.

In another embodiment, an adapted MFCC algorithm makes use of centralfrequencies of filters in a filter bank that are adjusted according to arobust measure of a central or mean frequency of the input audio, suchas the dominant frequency or spectral centroid. If this adapted MFCCalgorithm is used for fingerprinting of both query and reference audiosignals, the reference and query audio fingerprint comparisons can bemade more robust to pitch changes in the query audio signal.

In general, implementation of an MFCC algorithm facilitates parametricspectral representation of audio signals, which allows for generation ofmultidimensional audio descriptors with a plurality of dimensionsindependent of the number of filter bands. Quantization ofmultidimensional audio descriptor coefficients, where each coefficientrepresents a descriptor dimension, secures precise bit-wisemultidimensional audio signature generation for efficient databaseformation and search. These bit-wise multidimensional audio signaturescan be efficiently deployed as primary audio content signatures invarious application environments.

Input audio samples are divided into short, overlapping frames, andsubsequently windowed with a particular windowing function to preventartifacts in the output of an applied fast Fourier transform (FFT) dueto the finite extent of time samples. The amount of the consecutiveaudio frame overlap is determined in accordance with the desiredfrequency in time with which audio descriptors are generated. Next, theFFT is computed for each overlapped, windowed, audio frame, and then theresulting high-resolution frequency spectrum is used as an input to afilter bank.

The filter bank may suitably be an MFCC filter bank with logarithmicspacing of center frequencies, or in a presently preferred embodiment,it can be adjusted according to a robust measure of the central or meanfrequency of the input audio, such as the dominant frequency or spectralcentroid.

For the case of an adjusted center frequency filter bank, a centralfrequency estimate from the spectral magnitude of the current audioframe is made. For example, with known default center frequencies of theMFCC filter bank filters, one of the filters in the filter bank isdetermined whose center frequency is closest to a central-frequencymeasure previously established. A multiplier factor is generated toscale this filter's center frequency to the central-frequency measure,and the other filters center frequencies are multiplied by this factor.The MFCC coefficients which are the amplitudes of the resulting spectrumare then computed in accordance with the adjusted filter bank.

In both cases, an advantageous logarithm of the output from each filterof the filter bank is computed to handle a wider range of volume levels.Alternatively or in addition to the logarithm computation, anadvantageous square root (sqrt) of the output from each filter of thefilter bank is computed to handle higher levels of noise. Then, adiscrete cosine transform (DCT) is applied on the resulting signal toconvert the log and/or the sqrt outputs from the filter bank to a newset of values and frequencies. Next, an advantageous multi-tap smoothingderivative finite impulse response (FIR) filter is applied in temporaldomain on multiple audio descriptors which are outputs of the DCT stageof the computation computed in regular temporal intervals defined by thechosen amount of audio frame overlap. The multi-tap smoothing derivativeFIR filter is applied in temporal domain separately on each audiodescriptor coefficient, the DCT coefficient, to produce new, filteredDCT coefficients, representing a final multidimensional audio descriptoroutput.

FIG. 4A illustrates a process 400 for processing a video fragment, suchas the frame captured in step 401, acquired by a mobile camcorderoperating on the smart phone 110. A smartphone software application maybe utilized to provide a fingerprinting and search function. Video frame401 of the fragment is initially analyzed to identify a TV screen in theframe. In an exemplary case, the TV screen identification may be done byrunning edge detection on selected frames, by running corner detectionon selected frames, and searching for an initial quadrilateral ofappropriate dimensions as illustrated in step 402. A detectedquadrilateral is further refined by tracking motion from frame to frameof pixel formations inside and in the immediate neighborhood of thequadrilateral. Also, brightness and color of the detected quadrilateralcan be checked against the rest of a frame's content to further increaseconfidence that the TV screen area is correctly delineated. The framearea, corresponding to the detected quadrilateral, describing the TVscreen active area is cropped and warped to an upright rectangle in step403 and used for video fingerprint generation of the TV programmingcontent captured from the TV set. The smartphone application generatesquery fingerprints for the acquired fragment of media content andtransmits the query fingerprints to a search function. For example, asearch query may be sent to a local search function operating on themobile device or to a remote search server. The search function, eitherlocally or remotely, searches for the query content in a referencedatabase of the local mobile device and or an extensive referencedatabase of the remote search server.

FIG. 4B illustrates an exemplary and advantageous process 404 formultiple primary audio signature generation that targets mobile audiodistortions. At step 405, captured audio samples are received forprocessing. At step 406, input audio is down-sampled to mono audio andresampled to 14 kHz. At step 408, the mono audio is partitioned intooverlapping frames of size 4096 samples and a frame position incrementis specified as 256 samples, for example, resulting in a frame overlapof 3840 samples. At step 410, a Hamming window is applied on each inputframe of samples to reduce artifacts in FFT computation due to thefinite length of the input signal. At step 412, an FFT magnitude ofwindowed samples is generated, producing a 2048-point spectrum perframe, the spectrogram.

At step 414, the exemplary process 404 continues on to primary signatureA and signature B generation steps. Using the spectrogram, MFCCcoefficients are generated and processed in steps 420-426 for the twodistinct signatures A and B. At step 420, for signature A, a filter bankis applied on K frequency bands, such as K=24, producing K outputcoefficients divided into m linearly spaced bands across 200-1100 Hz,such as m=8, and n logarithmically spaced bands across 1100-6.4 kHz,such as n=16. Also, at step 420, a log₁₀ magnitude on the filter bankoutputs is computed. At step 422, a discrete cosine transform (DCT) iscomputed on the filter bank outputs to produce descriptor coefficients.At step 424, for signature B, a filter bank is applied on the samenumber K of frequency bands, producing K output coefficientslogarithmically spaced across 200-2 kHz. Also, at step 424, a squareroot (sqrt) function is applied on the filter bank outputs to producefinal filter bank outputs.

At step 426, a DCT is computed on the final filter bank outputs toproduce descriptor coefficients. Next, at steps 428 and 430, finaldescriptors A and B are derived by applying in temporal domain a 9-tapfinite impulse response (FIR) smoothing derivative filter to eachdimension of 9 consecutive descriptors computed in regular intervals,for example in intervals of 256 audio samples. This filter is appliedseparately on the coefficients of the set of A descriptors and on theset of B descriptors. The input to each filter consists of the currentvalue of the descriptor coefficient, which is also referred to as adimension, and the corresponding coefficients, also referred to asdimensions, from descriptors for the previous 8 audio frames. A set of 9FIR filter coefficients {h₀, h₁, h₂, h₃, h₄, h₅, h₆, h₇, h₈} aredesigned to produce a smoothing derivative filter response for eachdescriptor coefficient or dimension. The filter coefficients areanti-symmetric, generated in the (−1, 1) interval.

At step 432, the descriptor coefficients are quantized to either 0 or 1based on the coefficient sign. If the descriptor coefficient is greaterthan 0 a value of 1 is assigned to it, and if the descriptor coefficientis less than 0 a value of zero is assigned to it. The quantized valuesfor each descriptor coefficient are concatenated together to produce a24-bit signature. Signatures are then selected by choosing onlysignatures with at least k zeros and k ones, wherein k is apredetermined value. Signatures with fewer zeros or ones are suppressed.At step 434, filtered primary signatures A and B are output to thesignature selection and database formation process 1123, as shown inFIG. 11B.

FIG. 5 illustrates a process 500 for generation of audio clustersignatures, also known as traversal hash signatures. A number of lowercoefficients of primary descriptors are extracted and then combined withadditional bits from a set of independently detected audio contentfeatures to form the audio cluster signatures. At steps 504, primarysignatures A and B are received as inputs to the process 500. At step506, the lower 16 coefficients are extracted from both the primarysignatures A and B producing two 16-bit signatures that are subsets ofsignatures A and B. At step 508, an additional 4 bits are produced foreach subset of signatures A and B from selected additional audiofeatures, as described in more detail below. The additional 4 audiofeature signature bits are concatenated to their respective signatures Aand B, resulting in two 20-bit cluster signatures. At step 510, a bit isadded to both cluster signatures A and B; a 0 for cluster signature Aand a 1 for cluster signature B. At step 512, two 21-bit clustersignatures are sent to signature selection and database formationprocess 1123 of FIG. 11B, with the exemplary data structure andsignature format shown in FIG. 11A. The 21-bit cluster signature is usedas an address for reference database partitioning into clusters ofsignatures. The 21-bit cluster signature address facilitates a fastaccess of content in the reference database during a content query andidentification process. Selection of 16-bits from the primary signature,the four audio feature signature bits from generated audio features, andthe formation of a 21-bit cluster signature as shown in the process 500is exemplary. Other reasonable numbers of bits could be selected that incombination are less than the number of signature bits generated for thevideo frame 403, for example, such as the exemplary 24-bit primarysignatures A and B generated at step 432 of FIG. 4B. It is noted thatthe exemplary 24-bit primary signatures A and B may be expanded byconcatenating a number of feature bits resulting in 32-bit primarysignatures, as described in more detail below. It is also noted that analternative process may generate primary signatures A and B having forexample 64-bits. In any of these cases, having 21-bit cluster signaturesthat are used as addresses facilitates a fast access of content in thereference database during a content query and identification process.

It is noted that in a controlled experimental environment, audio framesextracted from an audio signal are aligned to a multiple of frame stepsize, typically 256 or 512 samples, with a regular interval. However, ina real life dynamic environment, a starting point of the audio frames inthe reference and query are generally randomly positioned with referenceto each other. Hence, it would be advantageous if audio frames betweenthe reference and the query signal are aligned based on some intrinsicaudio signal features.

In another embodiment, audio signal onset detection in the time domainis used for audio frame alignment. Audio signal onset detection is anaudio analysis technique that can be used to improve a fingerprintingsystem by aligning an audio signal fingerprinting window to an onsetevent. Onset events can also be used for feature signature or clustersignature bit generation, as well as for combining pairs of signaturesbased on distinct onset event times. If the same onsets are found onboth the reference and the query audio signals, audio frames will bealigned to the same audio content on both the reference and the querysignals. If a gap between detected onsets is larger than a frame stepsize, then additional audio frames can be generated with a regularinterval relative to the onset. Also, by using audio frames when onsetsare found, the number of audio signatures generated can be reduced.

FIG. 6 illustrates a process 600 for audio signal onset detection in thetime domain. The process 600 is used to improve the audio fingerprintingand identification system, such as the system 100 of FIG. 1, by aligningaudio signal fingerprinting windows to pronounced events intrinsic tothe audio signal. The process 600 is a temporal process for onsetdetection that is precise and computationally efficient. At step 604, amulti-channel audio waveform is received and then down mixed to a monosignal, and resampled to a desired sampling rate. At step 606, an energywaveform of the audio signal is computed by squaring the resampled audiowaveform. At step 608, a low-pass filter is applied to the energywaveform signal and resampled, for example to approximately 1 ms. Atstep 610, a filtered derivative of the resampled filtered energywaveform signal is computed for a multitude of derivative filter widths,to produce filtered derivative signals. Next, at step 612, maximumvalues of the filtered derivative signal are computed for a multitude ofmaximum filter widths. At step 614, maximum values that exceed aheuristic threshold are collected. At step 616, time positions ofmaximum values that exceed the threshold are output as a set of onsetsfor the current segment of audio signal.

The filtered derivatives of the low passed energy audio signal computedat step 610 represent a type of 1-D blob detector over the receivedaudio waveform. By varying the derivative filter width at step 610 andthe maximum filter width at step 612, audio signal onsets at differentpoints in time are obtained.

FIG. 7 illustrates a process 700 for utilization of audio signal onsetinformation. At step 704, a multichannel audio signal is received. Instep 706, the multichannel audio signal is down mixed to a mono signalwhich is then resampled as needed. For example, the received audiosignal may generally be sampled with 48 KHz or 44 KHz and the monosignal may be resampled to 14 KHz, however, the necessity of downsampling may be determined in a particular implementation. At step 708,output onset parameters are computed describing the time position ofonset events and onset magnitudes. At step 710, an onset position isused to align an audio signal fingerprinting window to a selected onsetevent. At step 712, for each audio frame, the time distance to theprevious onset, and separately to the next onset are computed. Thecombined time distance to the previous onset and to the next onset istermed an “onset-offset” value. At step 714, the onset-offset value isquantized to a k-bit value to be used as an independent featuresignature in the search and correlation process for audio contentidentification. In step 716, the onset-offset value is quantized to anm-bit value to be used as additional bits concatenated to the clustersignatures. At step 718, the output from steps 710, 714, and 716 arecollected and passed to signature A and signature B selection anddatabase formation process 1123 of FIG. 11B.

Three exemplary embodiments for audio content feature extraction andfeature signature generation are described next. A method for timedomain audio frame onset detection is described with regard to FIG. 8, amethod for frequency domain entropy computation for each audio frame isdescribed with regard to FIG. 9, and a method for extraction of maximumdifference in the output descriptor coefficients is described withregard to FIG. 10. The feature signatures, as described above, are usedeither independently during the correlation stage of the audio searchand identification, or selected bits from the feature signatures arecombined and concatenated with either primary or cluster signatures orboth signatures to improve the uniqueness of primary and clustersignatures.

FIG. 8 illustrates an exemplary process 800 for onset detection in thetime domain based on audio frame envelope tracking and continuous movingaverage tracking. The array of onsets computed with this process foreach audio frame can be used as an audio frame feature to be associatedwith the audio frame primary or cluster signature. At step 804, eachaudio frame received for processing is divided into 16 segments of 256samples. At step 806, an audio signal envelope detector is used to tracean audio signal envelope. The envelope detector algorithm includesvariable attack and release times that can be independently adjustedaccording to a desired sensitivity to audio signal content changes. Foreach audio frame, a value of the envelope is recorded at the end of each256 sample segment. At step 808, a moving average with an exponentialdecay is computed continuously over the same audio frame. At step 810,at the end of each 256 sample segment, the envelope value determined atstep 806 is compared to the moving average value determined at step 808,and a difference vector is generated at step 810. At step 812, thedifference vector L₂ norm is computed. At step 814, the differencevector elements are normalized to the difference vector L₂ norm. At step816, a maximum difference vector element is determined and normalized.At step 818, the normalized maximum difference vector element from step816 is quantized to a 4 bit value and its position within the audioframe is recorded. At step 820, the normalized maximum difference vectorelement and its position are delivered as audio feature signatures tothe signature selection and database formation process 1123 of FIG. 11B,with the exemplary signature data structure 1100 as shown in FIG. 11A.

FIG. 9 illustrates a process 900 for an exemplary audio frame frequencydomain entropy computation as an audio frame feature to be associatedwith the audio frame primary and cluster signatures. At step 904, audioframes are received for processing. At step 906, an FFT is computed foran array of N received audio frames. At step 908, a spectral magnitudeof the FFT output from step 906 is used to compute an entropy functionfor spectral magnitude samples. An exemplary entropy function 908 may becomputed as follows. If M(f) is the magnitude of the spectrum forfrequency position fin a particular audio frame, then M_(s)(f) is M(f)divided by the sum: sum(M(f)), computed for the audio frame. Thespectral entropy is computed as a sum: sum{log₂[M_(s)(f)]*M_(s)(f)},where the summation is taken over the range of frequencies in the audioframe. At step 910, a first temporal derivative of the entropy iscomputed for each two consecutive audio frames. This temporal derivativecomputation corresponds to computing a sign of an entropy change betweentwo consecutive frames. At step 912, signs of the most recent N samplesof the entropy difference values from step 910 are concatenated to forman N-bit entropy sign difference history feature vector. At step 914,the N bit history feature vector is delivered as a feature signature tothe signature selection and database formation process 1123 of FIG. 11B,with the exemplary signature data structure 1100 as shown in FIG. 11A.

FIG. 10 illustrates a process 1000 to compute an exemplary position ofthe maximum difference in the audio descriptor coefficients between twoconsecutive audio frames, to be associated with the audio frame primaryor cluster signature as an audio frame feature. At step 1004, audiodescriptors from step 436 of FIG. 4B for two consecutive audio framesare received for processing. At step 1006 and for each audio frame, adifferential descriptor vector is computed. At step 1008, the largestmagnitude difference in the upper X dimensions, such as the upper 12dimensions as determined experimentally or by heuristic, of thedescriptor is determined. At step 1010, a 4 bit value is determined asrepresenting a position of the maximum difference within the X dimensiondifferential descriptor vector. At step 1012, the positional index valueis delivered as a feature signature to the signature selection anddatabase formation process 1123 of FIG. 11B, with the exemplarysignature data structure 1100 as shown in FIG. 11A.

FIG. 11A illustrates an exemplary audio-video signature data structure1100. Block 1104 represents a primary signature A, such as an exemplary24-bit value, which is concatenated with a feature aspect 1106, such asan exemplary 8-bit value, associated with the primary signature A. In asimilar manner, block 1108 represents a primary signature B, such as anexemplary 24-bit value, which is concatenated with a feature aspect1110, such as an exemplary 8-bit value, associated with the primarysignature B. Blocks 1112 and 1122 represent p-bit additional featuresused during the correlation process of audio content search andidentification to further differentiate matching results, where p may bean exemplary sixteen bit value. Also, feature bits may be added to theprimary signature to increase the signature length. A similar datastructure is formed for the cluster signature A, blocks 1114 and 1116,and cluster signature B, blocks 1118 and 1120.

As an example, additional q-bits in the cluster signatures may be formedas a mixture of selected bits from an onset feature, selected bits froman entropy feature, and selected bits from a maximum change in thedescriptor coefficients feature. Block 1114 represents a clustersignature A, such as an exemplary 16-bit value, which is concatenatedwith a q-bit feature aspect block 1116 associated with the clustersignature A, where q may be an exemplary 5-bit value. In a similarmanner, block 1118 represents a cluster signature B which isconcatenated with a q-bit feature aspect block 1120 associated with thecluster signature B. Features, primary signatures, and clustersignatures are packed into the signature data structure as shown in FIG.11A, and subsequently used for media search and identification, asdescribed in more detail below.

As presented above, primary and cluster audio signatures are formed as amixture of bits, representing dimensions of the associated signatures,selected from the MFCC filtered output, and additional audio featuresbits. Both combined K-dimension primary signature and combinedM-dimension cluster signature are generated for each audio featureidentified in a set of reference multimedia clips. Similarly, exemplaryL-dimension video primary signatures and N-dimension video clustersignatures, as well as x, y, and scale feature signatures, are formed asdescribed in U.S. Pat. No. 8,189,945 titled “Digital Video ContentFingerprinting Based on Scale Invariant Interest Region Detection withan Array of Anisotropic Filters” and U.S. Pat. No. 8,195,689 titled“Media Fingerprinting and Identification System” which are herebyincorporated by reference in their entirety.

An exemplary first audio and video fingerprinting process would includemultiple video signatures generated on an active TV area as shown in theprocessed video frame in step 403 of FIG. 4A, onset detection for audiofingerprint alignment, such as the onset detection process 600 of FIG.6, multiple primary audio signatures generated in process 404 of FIG.4B, multiple audio cluster signatures generated in process 500 of FIG.5, and feature signatures generated in processes 800 of FIG. 8, 900 ofFIG. 9, and 1000 of FIG. 10. The signatures in steps 432, 510, 818, 912,and 1010, are advantageously combined in the signature selection anddata base formation process 1123 of FIG. 11B. It is noted that thefeature signatures from steps 818, 912, and 1010 may be suitablycombined by concatenation or by another combinational method to generatea combined feature signature.

In another exemplary case, a second audio and video fingerprintingprocess would include onset detection for audio fingerprint alignment600, multiple primary audio signatures generated in process 404,multiple audio cluster signatures generated in process 500, and featuresignatures generated in process 700. The signatures in steps 434, 512,714, and 716 would be combined in the signature selection and databaseformation process 1123 of FIG. 11B.

It is noted that multiple exemplary combinations of signaturesgenerated, as illustrated in FIGS. 4A, 4B, 5, 6, 7, 8, 9, and 10,selection, and data base formation processes may be used. Anexperimental analysis may be used together with application requirementsto select or adapt the signature generation process most appropriate fora given situation.

Each K(L)-dimension signature and a link to a corresponding referencemultimedia clip are stored at a location in a reference signaturedatabase residing either on the remote server or in storage on the localmobile device. Each location is addressable by the M(N)-dimensioncluster signature, also described as a traversal hash signature. AK(L)-dimension query signature and an M(N)-dimension query clustersignature are generated for a query multimedia clip. The referencesignature database is searched using the query cluster signature to findsimilar signatures that are within a specified signature distance,wherein the similar reference multimedia clips are aggregated in acandidate list of closely matching signatures that correspond to similarreference multimedia clips. Additional feature signatures may also beused for media query and reference signature correlation to strengthenthe scoring process and reduce false positive media identification.

FIG. 11B illustrates a signature selection and database formationprocess 1123. In step 1124, the primary, cluster, and feature signaturesare received. In step 1126, primary signatures A and primary signature Bare selected based on multiple criteria related to a particularapplication requirement. In step 1128, feature signatures are selectedto form associated feature signatures. For example, the selection may bedependent on general computational requirements and the database sizelimitation. In step 1130 cluster signatures A and B are selected basedon the reference signature data base size, the required query responsetime, and the required query precision and recall. In step 1132, the setof primary, cluster, and feature signatures from steps 1126, 1128, and1130 are combined and then used to form an audio signature database instep 1134 with the exemplary signature data structure 1100 shown in FIG.11A.

FIG. 12 illustrates a process 1200 for fingerprinting and search withmultiple signatures and features. For example, multiple signatures andfeatures may be derived from TV program video content. In the process1200 of FIG. 12, it is assumed that the reference database has beenformed in advance by use of a fingerprinting process, such asincorporated in the database formation process 1123 of FIG. 11B. At step1204, a query audio signal, or a query video signal, or combined audioand video query signals are submitted for fingerprinting. At step 1206,primary, cluster and feature query signatures are generated. At step1208, query signatures are generated for a chunk of query content,usually 5-10 seconds worth of query fingerprints, and subsequently usedfor content identification. At step 1214, a cluster search is performedto determine a reference signature cluster closest to the query clustersignature. At step 1216, a similarity search is conducted with the queryprimary signature on the primary signatures belonging to the identifiedcluster. Search result scores are generated at step 1216 and matchingcandidates from the reference content are identified. At step 1210,query feature signatures are compared to the reference featuresignatures for all candidates belonging to the identified cluster, andat step 1212, a feature score is generated. At step 1220, primary andfeature signature scores are combined for all candidate matches. At step1222, the resulting score is compared against a threshold T. If amatching reference audio score is above the threshold T, the process1200 proceeds to step 1226 where notification is presented of asuccessful search result. At step 1228, a software application which canperform actions that are synchronized to the video content aretriggered. If all candidate matching scores are below the threshold T,the process 1200 proceeds to step 1224 where notification is presentedof a not found result. The threshold T, used for signature and featurecomparison score evaluation, may be heuristically generated or generatedbased on a training process on an extensive training database.

FIG. 13 illustrates a process 1300 for combining audio and video database search results as generated in the process 1200 to refine thesearch process and improve accuracy in finding a matching audio andvideo sequence.

The audio and video database search results, such as a set of scores forcandidate matching audio and matching video sequences, are combined andfurther analyzed in the steps of process 1300 of FIG. 13. At step 1302,audio and video scores are received for a combined analysis. At step1304, search result scores and computed confidence levels for both audioand video search results are analyzed on multiple reported, in otherwords identified, top candidate audio and video titles. At step 1306, acombined audio and video score is generated. In step 1308, the combinedaudio and video score is tested against a threshold T1. If the combinedscore is below the threshold T1, an audio and video sequence not foundresult is reported at step 1310. If the combined score is greater thanthe threshold T1, an audio and video sequence found result is reportedat step 1312.

A method is presented for efficient and accurate actionable televisionevent identification based on television audio-video contentfingerprinting following various viewing scenarios in the broadcast andcable television transmission environment. An actionable media event isan event that is suitable to be acted on. Based on a detected actionableevent in a local set top box (STB) or properly equipped television setwith broadband connection, an enhanced television viewing functionalityis delivered via MSO installed equipment on top of current lineartelevision transmission services. In one scenario, when a TV program isviewed, contiguous segments of played video are collected in real time,fingerprinted, and subsequently used as queries into a locally preloadedreference fingerprint database representing a collection of knownadvertising clips. A successful ad identification is used as anactionable event, triggering various additional local advertising and/orvarious enhancement programming made available to the user.

In another scenario, a query video clip such as an ad, or a part of atelevision program, is fingerprinted in a remote location andsubsequently delivered through the Internet network to a local STB orproperly equipped TV set. Remotely delivered query fingerprints are usedto identify timely reference content of played television programscollected locally within an STB or a properly equipped TV set withbroadband connectivity. Fingerprinted reference television content isstored on a push down stack within an STB or properly equipped TV set,and is continuously refreshed with new television content. This scenariois used for fine granularity, fast verification of ads or othertelevision programming, presented to viewers within a period of timeequivalent to the duration of the locally collected TV programming. Fastdetailed audience measurements concerning a particular televisionprogram may be generated based on snippets, short clips, of remotelydelivered query television content fingerprints, subsequently identifiedat a locally generated short, timely reference database of fingerprintedtelevision content.

The system presented here includes a localized preloaded ad referencedatabase, collected in a real time television content referencedatabase, remote and local query generation, near real time remote andlocal query responses, and scalability across a multitude of devicesfrom mobile phones and TV STBs to TV sets upgraded with Internetprotocol (IP) connectivity. The system as described herein sets itselfapart from other systems in terms of accuracy, scale, and speed. Locallyaccumulated, limited amount of played television content stored in apush down stack in a local memory device, for example, allows fordetection of a played advertisement or other content by means of aremotely delivered advertisement query or other content query. It alsoallows querying a reference database locally preloaded withfingerprinted ads of choice, with fingerprinted short real timesequences of television content, and in a fraction of the ad duration,leading to subsequent changes in user device activity, including acoherent system of discovery, recommendation and new ad insertion.

Audio and video content fingerprinting for actionable televisionprogramming detection and identification presented herein provides formanaging content identification and detecting actionable events withhigh accuracy, simultaneously allowing for low cost design and easysystem upgrades. A service that provides a seamless and coherentenhanced user experience generally should also be based on accurateinformation about the viewer's activity. The existing relationshipbetween customers and MSOs and satellite connection providers creates atremendous advantage in using innovative technology to allow viewers thefreedom of exploring new possibilities, while maintaining the existingservices of MSOs and satellite connection providers. Also, astandardized user interface may be created to reliably identify a widevariety of video clips such as ads and other programming content withinpredetermined time periods when actionable events may be detected.

The potential of interactive, actionable TV may be achieved by having anaccurate identification of played content at the granularity of smallvideo-audio clips, such as ads and excerpts from the played content.

As used herein, a video fingerprint is a calculated numerical value,alternatively described as a video signature, which represents aprominent feature of a video frame. A collection of signatures drawnfrom a single video frame with multiple features generates a compactvideo frame representation. A collection of signatures associated withmultiple video frames in a video sequence forms a video sequencerepresentation. In addition, an audio fingerprint is a calculatednumerical value, alternatively termed an audio signature, whichrepresents a prominent feature of an audio frame. An audio frame is acollection of audio samples within a time window covering a portion ofan audio sequence. A collection of audio signatures associated withaudio frames in an audio sequence with multiple frames forms an audiosequence representation.

Video fingerprints are designed for a particular type of utilization,supporting invariance to various video production processing effectssuch as aspect ratio conversion, including 16×9 to 4×3 letterbox, or 4×3to 16×9 conversion, cross conversion between 720p and 1080i, up-downconversion from high definition (HD) to standard definition (SD), apercentage of brightness and contrast change, various compressiontechniques, and the like.

Audio fingerprints are designed to be robust to level changes such thatdifferent audio levels have minimal effect on the audio signature andchanges, such as stereo to mono down mix, 5.1 channel down mix tostereo, various encoding or decoding processes such as MPEG, Dolby-x,MP3, various other distortions, such as speed up or 5 slow down, pitchchange, gain change, noise, or the like.

The system architecture is based on a distributed audio and videofingerprint generation with a centralized or distributed monitoring andcontrol functions. Two separate reference fingerprint databases aremaintained. One set of reference television program content fingerprintsare generated at a distributed array of STBs or Internet protocol (IP)equipped television sets located at viewers' premises. For this case, areference fingerprinting process continuously fingerprints current TVcontent and subsequently refreshes a local reference fingerprintrevolving buffer of programmable length according to a predetermined,programmable schedule. In addition to reference fingerprints, relevantmetadata, such as program title, program length, close captioning, 15current date, and the like, associated with the fingerprinted TV contentare recorded. The reference database generated and refreshed asdescribed above is termed a reference television (ReferenceTV) database.

In another scenario, a set of reference fingerprints of selected videosequences are transferred to a distributed array of client mediadevices, such as an array of STBs or equipped television sets, to form areference database that is stored locally on each client media device.The selected video sequences may include a collection of currentlyrelevant advertising clips, which may be audio advertisement clips orvideo advertisement clips. The reference database on each client mediadevice is termed a reference advertisement (ReferenceAD) database.

In one embodiment of this invention, query fingerprints, representingshort known media sequences, such as advertising clips that aregenerally the beginning frames of an advertisement, may be generated ata remote location and transferred to selected client media devices. Aclient media device may be a digital radio, an IP equipped TV, a set topbox, a network attached game machine device, or the like. The queryfingerprints may also be broadcast to an array of client media devices,such as STBs and IP equipped television sets, or to a part of the arrayof STBs and IP equipped television sets over a TCP/IP network, forexample. The query fingerprints representing multiple different knownmedia sequences may be received into a local query buffer of the clientmedia device. The received query fingerprints are then periodically usedto search the current content of the reference media database, such asthe Reference TV fingerprint database stored, for example in aReferenceTV revolving buffer. A successful video clip identification isused as an actionable event, triggering various additional localadvertising and various enhancement programming made available to theuser. For example, enhancement programming may include information on apreviously viewed or currently viewed media selection in addition towhat the media selection generally provides, such as release dates,characters in a video sequence, title of an audio sequence, links toadditional information, related media content, and the like. Remotequery content fingerprinting, packetizing and delivery scheduling may bedifferent for different segments of the array of STBs and equippedtelevision sets and may be tailored to the target client media devices.

In another embodiment of this invention, short contiguous snippets ofcurrently played television program advertisements are collected andfingerprinted in real time, and subsequently used as queries into alocally preloaded ReferenceAD fingerprint data base representing acollection of relevant advertising clips. A successful ad identificationis used as an actionable event, triggering various additional localadvertising and various enhancement programming made available to theuser. The ReferenceAD database is updated according to a location of theclient media device and time of day of playing media content. Forexample, the ReferenceAD database may be updated with one set ofadvertisement fingerprints for advertisements shown in the morning and adifferent set of advertisement fingerprints for advertisements shown inthe evening. Also, the advertisements shown in a western state may bedifferent than an advertisement shown in an eastern state, with theclient devices in each location updated with the appropriate localadvertisement fingerprints.

FIG. 14 illustrates a system 1400 for distributed television contentaudio-video fingerprinting with a localized or distributed monitoringpoint, and a distributed array of STBs and IP equipped television sets1410, 1450, 1422, 1436, in accordance with an embodiment of the presentinvention. As shown in FIG. 14, one aspect of the process of actionableevent generation based on television program audio and video contentfingerprinting consists of generation of query video clip fingerprintsat a remote location, such as remote locations 1402-1408. The generatedquery fingerprints are transferred over a network, such as the Internet1420, to an array of STBs 1412, 1414, 1416, 1418, 1424, 1448, 1452,1454, 1456, 1458 and IP equipped television sets 1426-1434, 1438-1446.Each of the STBs and IP TVs subsequently monitors, detects andidentifies reference television content corresponding to a querycontent. In another aspect of the system presented in FIG. 1, queryfingerprints may be preloaded over the Internet 1420 into a local STB1412, 1414, 1416, 1418, 1424, 1448, 1452, 1454, 1456, 1458 or an IPequipped television set 1426-1434, 1438-1446 and then used continuouslyor periodically for detection and identification of the referencetelevision content corresponding to one or more of preloaded querycontents. Access to the array of STBs 1412, 1414, 1416, 1418, 1424,1448, 1452, 1454, 1456, 1458 and IP equipped television sets 1426-1434,1438-1446 through the Internet 1420 is also utilized to preload aReferenceAD 1516 database shown in FIG. 15.

Television content, part of the television content, and an ad areidentified by a local correlation process which compares referencefingerprints, contained in a reference fingerprint database, such as theReferenceTV database 1514, to query fingerprints loaded from theInternet 1420 into an STB or an IP equipped television set 1426-1434,1438-1446. Also, query fingerprints selected from a local query bufferresiding in an STB 1412, 1414, 1416, 1418, 1424, 1448, 1452, 1454, 1456,1458 or an IP equipped television set 1426-1434, 1438-1446 may becompared to the ReferenceTV database 1514 of FIG. 15 to identifytelevision content, part of the television content, an ad or a part ofan ad. If query fingerprints match played reference television contentfingerprints, a trigger point is generated to the application interface,prompting further actions handled by the application interface.

In another scenario, an ad is identified by a local correlation processwhich compares reference fingerprints, contained in the preloadedReferenceAD database 1516 of FIG. 15, to query fingerprints generatedfrom short contiguous snippets of played television content that aretemporarily saved and fingerprinted in real time. Successful adidentification is used as an actionable event that is passed through alocal application interface to an application that manages variousadditional local advertising and various enhancement programming.

FIG. 15 presents a system view 1500 of a reference point with a clientmedia device 1510, such as an MSO device, satellite STB device, an IPequipped television set, or a remote device 1502, in accordance with anembodiment of the present invention. The system 1500 generates anddelivers query fingerprints over a TCP/IP network using a mediacommunication device 1526. The client media device 1510 containsReference TV database 1514 and ReferenceAD database 1516, actionableevent detection utility 1518, currently played audio and video referencecontent fingerprinting utilities 1520, and command, control, data andstatistics preparation, and delivery utility 1522.

The remote device 1502 may also be suitably employed as a remotereference and query fingerprinting, client monitoring and data orstatistics collection unit. The remote device 1502 contains functionsfor audio and video content fingerprinting 1504, query and referencefingerprint packetization and delivery 1506, and remote clientmonitoring and data collection 1508. The remote device 1502 deliversreference and query fingerprints to the remote clients, monitors clientdevices and collects data and statistics from the client media devices1510.

The media communication device 1526 provides an interface between theclient media device 1510 and the remote device 1502. For example, themedia communication device 1526 may include a reference and queryaudio-video (AN) fingerprint delivery media function and a remote clientmonitoring and statistics collection function for communicating with theclient media devices through a network, such as the Internet, by using acommunication protocol, such as the TCP/IP protocol.

An exemplary embodiment of signature formation, also referred to asfingerprinting, and database formation is described in U.S. applicationSer. No. 12/141,163 filed Jun. 18, 2008, FIGS. 11-16 and page 25, line 3to page 28, line 18. Another exemplary embodiment of fingerprinting anddatabase formation is described in U.S. application Ser. No. 12/612,729filed Nov. 5, 2009, FIGS. 12-14 and page 28, line 20 to page 31, line13. Another exemplary embodiment of fingerprinting and databaseformation is described in U.S. application Ser. No. 12/491,896 filedJun. 25, 2009, FIGS. 8-10 and page 20, line 8 to page 24, line 22.Another exemplary embodiment of fingerprinting and database formation isdescribed in U.S. application Ser. No. 10 12/772,566 filed May 3, 2010,FIGS. 4-9B and page 23, line 6 to page 40, line 6. A further exemplaryembodiment of fingerprinting and database formation is described in U.S.application Ser. No. 12/955,416 filed Nov. 29, 2010, FIGS. 6-12, andpage 16, line 8 to page 29, line 15. An exemplary embodiment of a systemand database formation process is described in U.S. application Ser. No.12/772,566 filed May 3, 2010, FIGS. 1-3, and page 10, line 10 to page23, line 5. An exemplary embodiment of query search is described in U.S.application Ser. No. 12/141,163 filed June 18, 2008, FIG. 17, and page28, line 19 to page 29, line 6. Another exemplary embodiment of querysearch is described in U.S. application Ser. No. 12/612,729 filed Nov.5, 2009, FIG. 15, and page 31, line 14 to page 32, line 15. A furtherexemplary embodiment of query search is described in U.S. applicationSer. No. 12/772,566 filed May 3, 2010, FIGS. 10-13, and page 40, line 7to page 45, line 14. Modifications of the above illustrative approachesor other approaches may be employed consistent with the teachings of thepresent invention.

FIG. 16 illustrates a process 1600 for audio and video fingerprinting atclient devices such as an STB or an IP TV set, and reference televisioncontent database formation in accordance with an embodiment of thepresent invention. In FIG. 16, real time television content audio andvideo fingerprinting is accomplished at the client media device 1510,and used to form a subsequent audio and video ReferenceTV fingerprintdatabase 1514. Reference television programming content, representingchunks of original material, is fingerprinted on the fly, while thetelevision programming content is being presented to the viewer. Anumber of chunks of the current reference television contentfingerprints are stored on a push down stack, thus forming theReferenceTV data base 214. The Reference TV database 1514 iscontinuously refreshed with new chunks of fingerprints derived from thecurrently playing television programming content while the oldest chunksare removed from the database.

In FIG. 16 the process 1600 starts with a television program, played atan STB device or Internet protocol (IP) equipped TV set at step 1602.The original content is broken into contiguous, or noncontiguous, inputchunks of uncompressed content at step 1604 for further processing. Theinput program content chunks from step 1604 are processed in step 1606for video active region of interest determination and frame selection.The input chunks from step 1604 are also processed in step 1608 as wellfor audio framing into predetermined segment sizes. Next and inparallel, video content fingerprints are generated in step 1610, andaudio content fingerprints are generated in step 1612. In step 1614,audio and video fingerprints are added to the ReferenceTV database 1514.In step 1616, a determination is made whether the ReferenceTV database1514 is full. If the Reference TV database 1514 is full, the oldestchunk of fingerprints are removed from the ReferenceTV fingerprintdatabase 1514 at step 1618 to make sufficient room for newly generatedfingerprints. The process 1600 then returns to step 1604. Returning tostep 1616, if the Reference TV database 1514 is not full, the process1600 returns to step 1604.

FIG. 17 illustrates a process 1700 for video snippet fingerprinting,preloaded reference ad data base search, and actionable event generationin accordance with an embodiment of the present invention. In FIG. 17,the process 1700 generates short contiguous snippets of currently playedvideo collection and real time fingerprints, which are subsequently usedas queries into the locally preloaded ReferenceAD database 1516. TheReferenceAD database 1516, as stated above, represents a collection ofrelevant advertising clips preloaded by the remote reference and queryfingerprinting, client monitoring and data or statistics collectionremote device 1502. The process 1700 of FIG. 17 starts at step 1702 witha television program, played at STB device or equipped TV set. At step1704, the played content is broken into contiguous, or noncontiguous,input chunks of uncompressed content for further processing. At step1706, the input program content chunks from step 1704 are processed forvideo active region of interest determination and frame selection. Atstep 1708 and in parallel with step 1706, audio framing processes audiocontent into predetermined segment sizes. Next, video contentfingerprints are generated in step 1710, and in parallel audio contentfingerprints are generated in step 1712. In step 1714, audio and videofingerprints are searched against the preloaded ReferenceAD database1516. In step 1716, a decision is made whether the query fingerprintscorrespond to any fingerprints in the Reference AD database 1516 offingerprints. If there is a correspondence between query fingerprintsand reference fingerprints, an ad has been detected. Successful adidentification may occur in a fraction of the ad duration, before theentire ad has been played. Ad identification is subsequently used as anactionable event in step 1718. Upon detecting an actionable event anindication of the event is passed to a local application, which may, forexample, replace the currently played ad with a locally, regionallyrelevant ad. The actionable event generated at step 1718 may overlay thecurrently played ad with additional information important to the local,regional community. Upon reception of an actionable event, the localapplication may also present a choice of additional programming channelsand programming enhancements, offered as a menu overlaid on top of thecurrent programming. After step 1718, the process 1700 proceeds to step1704. Returning to step 1716, if an ad has not been detected, theprocess 1700 also proceeds to step 1704.

FIG. 18 illustrates a process 1800 for remote querying of the referenceTV database, and actionable event generation based on recently playedtelevision content identification in accordance with an embodiment ofthe present invention. Beginning at step 1802, an Internet TCP/IPconnection is monitored for a desired opening of a communication channelin response to a request for communication. If a request forcommunications between the remote device 1502 and the client mediadevice 1510 is detected on either side, remote query fingerprints thatare available or remote reference ads that are available, or both aretransferred over the Internet TCP/IP communication channel at step 1804.If the remote query fingerprints are not available and if the remotereference ads are not available, the process 1800 proceeds back to step1802. The remote queries or ads are loaded into the local client mediadevice 1510 in step 1806. Subsequently, in step 1808, the localReferenceTV database 1514 is searched with remotely loaded queries forplayed video content identification. If reference television content isidentified by the search, such as by the search module 212, anactionable event is detected by the actionable event detection utility1518 which is generally a local application, a locally prescribedactionable event is generated in step 1810. The actionable event maythen trigger a plurality of additional viewing experiences offered tothe viewer by the local application 1518. Upon reception of a triggerevent, the local application 1518 may deliver to a remote monitoringlocation various statistics 222 related to the viewed televisioncontent, including user behavior and preferences. A collection ofstatistics from the array of STB devices and IP TV sets 1510, 1422,1436, 1450 may be used at the remote monitoring location for identifyingan audience for a particular television program. Through the localapplication 1518, an actionable event may also trigger local advertisingoverlaid on top of the current program and it may present a choice ofadditional programming channels offered as a menu overlaid on top of thecurrent programming.

It is understood that other embodiments of the present invention willbecome readily apparent to those skilled in the art from the followingdetailed description, wherein various embodiments of the invention areshown and described by way of the illustrations. As will be realized,the invention is capable of other and different embodiments and itsseveral details are capable of modification in various other respects,all without departing from the present invention. Accordingly, thedrawings and detailed description are to be regarded as illustrative innature and not as restrictive.

1. A computer-implemented method comprising: identifying, by a computingdevice, a client media device of an array of client media devices;determining, by the computing device, a location of the client mediadevice; based on the location of the client media device and a time ofday, selecting a set of advertisement fingerprints from among aplurality of advertisement fingerprints, wherein advertisementfingerprints of the set of advertisement fingerprints correspond torespective known advertisements presented during the time of day; andtransmitting, by the computing device to the client media device over anetwork, the set of advertisement fingerprints for comparison withreference fingerprints in a reference database of the client mediadevice.
 2. The computer-implemented method of claim 1, furthercomprising receiving, from the client media device, an indication thatthe client media device presented a known advertisement with which anadvertisement fingerprint of the set of advertisement fingerprintscorresponds.
 3. The computer-implemented method of claim 2, furthercomprising identifying an audience for a television program based on theindication and a collection of indications received from other clientmedia devices of the array of client media devices.
 4. Thecomputer-implemented method of claim 1, further comprising: based on thelocation of the client media device and another time of day, selectinganother set of advertisement fingerprints from among the plurality ofadvertisement fingerprints, wherein advertisement fingerprints of theother set of advertisement fingerprints correspond to respective knownadvertisements presented during the other time of day; and transmitting,to the client media device over the network, the other set ofadvertisement fingerprints.
 5. The computer-implemented method of claim1, further comprising generating the plurality of advertisementfingerprints using respective frames of the known advertisements.
 6. Thecomputer-implemented method of claim 1, wherein determining the locationof the client media device comprises determining that the client mediadevice is located in a first geographic region rather than in a secondgeographic region.
 7. The computer-implemented method of claim 1,wherein the reference database comprises a revolving buffer of aprogrammable length.
 8. The computer-implemented method of claim 1,wherein the client media device is a network-connected client videodevice, and wherein the advertisement fingerprints comprise videofingerprints.
 9. The computer-implemented method of claim 1, wherein theclient media device is a network-connected client audio device, andwherein the advertisement fingerprints comprise audio fingerprints. 10.A computing device comprising: one or more processors; and acomputer-readable medium having stored therein instructions that areexecutable by the one or more computers to cause the computing device toperform functions comprising: identifying a client media device of anarray of client media devices, determining a location of the clientmedia device, based on the location of the client media device and atime of day, selecting a set of advertisement fingerprints from among aplurality of advertisement fingerprints, wherein advertisementfingerprints of the set of advertisement fingerprints correspond torespective known advertisements presented during the time of day, andtransmitting, to the client media device over a network, the set ofadvertisement fingerprints for comparison with reference fingerprints ina reference database of the client media device.
 11. The computingdevice of claim 10, wherein the functions further comprise receiving,from the client media device, an indication that the client media devicepresented a known advertisement with which an advertisement fingerprintof the set of advertisement fingerprints corresponds.
 12. The computingdevice of claim 11, wherein the functions further comprise identifyingan audience for a television program based on the indication and acollection of indications received from other client media devices ofthe array of client media devices.
 13. The computing device of claim 10,wherein the functions further comprise: based on the location of theclient media device and another time of day, selecting another set ofadvertisement fingerprints from among the plurality of advertisementfingerprints, wherein advertisement fingerprints of the other set ofadvertisement fingerprints correspond to respective known advertisementspresented during the other time of day; and transmitting, to the clientmedia device over the network, the other set of advertisementfingerprints.
 14. The computing device of claim 10, wherein thefunctions further comprise generating the plurality of advertisementfingerprints using respective frames of the known advertisements. 15.The computing device of claim 10, wherein determining the location ofthe client media device comprises determining that the client mediadevice is located in a first geographic region rather than in a secondgeographic region.
 16. The computing device of claim 10, wherein thereference database comprises a revolving buffer of a programmablelength.
 17. A non-transitory computer-readable medium having storedtherein instructions that are executable by one or more processors tocause a computing device to perform functions comprising: identifying aclient media device of an array of client media devices; determining alocation of the client media device; based on the location of the clientmedia device and a time of day, selecting a set of advertisementfingerprints from among a plurality of advertisement fingerprints,wherein advertisement fingerprints of the set of advertisementfingerprints correspond to respective known advertisements presentedduring the time of day; and transmitting, to the client media deviceover a network, the set of advertisement fingerprints for comparisonwith reference fingerprints in a reference database of the client mediadevice.
 18. The non-transitory computer-readable medium of claim 17,wherein the functions further comprise receiving, from the client mediadevice, an indication that the client media device presented a knownadvertisement with which an advertisement fingerprint of the set ofadvertisement fingerprints corresponds.
 19. The non-transitorycomputer-readable medium of claim 18, wherein the functions furthercomprise identifying an audience for a television program based on theindication and a collection of indications received from other clientmedia devices of the array of client media devices.
 20. Thenon-transitory computer-readable medium of claim 17, wherein thefunctions further comprise: based on the location of the client mediadevice and another time of day, selecting another set of advertisementfingerprints from among the plurality of advertisement fingerprints,wherein advertisement fingerprints of the other set of advertisementfingerprints correspond to respective known advertisements presentedduring the other time of day; and transmitting, to the client mediadevice over the network, the other set of advertisement fingerprints.